{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "82a86cb2-ea9d-43f1-977b-72e5cc5a708b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:00.374035Z",
     "iopub.status.busy": "2024-10-27T14:42:00.373156Z",
     "iopub.status.idle": "2024-10-27T14:42:00.389731Z",
     "shell.execute_reply": "2024-10-27T14:42:00.388173Z",
     "shell.execute_reply.started": "2024-10-27T14:42:00.373987Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['http_proxy'] = 'http://127.0.0.1:7890'\n",
    "os.environ['https_proxy'] = 'http://127.0.0.1:7890'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed8cf62b-6270-403b-adf2-89a5d5fcf197",
   "metadata": {},
   "source": [
    "### GCN => GAT "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "910443d0-8554-4864-9a86-e4a8ce65eda4",
   "metadata": {},
   "source": [
    "- GCN：Graph Conv Networks, GAT：Graph Attention Networks\n",
    "- 都是用来学习 target node 的 representation\n",
    "- gcn\n",
    "    - $d_v$：表示节点的degree（即有多少条边连接），注意包含自己；\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{h}_{\\mathcal{N}(v)} &= \\sum_{u \\in \\mathcal{N}(v)} w_{u,v} \\mathbf{h}_u \\\\\n",
    "&= \\sum_{u \\in \\mathcal{N}(v)} \\sqrt{\\frac{1}{d_v}} \\sqrt{\\frac{1}{d_u}} \\mathbf{h}_u \\\\\n",
    "&= \\sqrt{\\frac{1}{d_v}} \\sum_{u \\in \\mathcal{N}(v)} \\sqrt{\\frac{1}{d_u}} \\mathbf{h}_u\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "- gat\n",
    "    - https://www.youtube.com/watch?v=SnRfBfXwLuY\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{h}_{\\mathcal{N}(v)} &= \\sum_{u \\in \\mathcal{N}(v)} \\underbrace{\\text{softmax}_u \\left( a(\\mathbf{h}_u, \\mathbf{h}_v) \\right)}_{\\alpha_{u,v}} \\mathbf{h}_u \\\\\n",
    "\\alpha_{u,v} &= \\frac{\\exp\\left(a(\\mathbf{h}_u, \\mathbf{h}_v)\\right)}{\\sum_{k \\in \\mathcal{N}(v)} \\exp\\left(a(\\mathbf{h}_k, \\mathbf{h}_v)\\right)}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9c8d05-6dad-4104-a786-8c35a501081d",
   "metadata": {},
   "source": [
    "### GAT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d37a53c6-8bc4-4ec0-802e-b17b2623b69b",
   "metadata": {},
   "source": [
    "https://github.com/AntonioLonga/PytorchGeometricTutorial/blob/main/Tutorial3/Tutorial3.pdf\n",
    "- https://github.com/AntonioLonga/PytorchGeometricTutorial/blob/main/Tutorial3/Tutorial3.ipynb\n",
    "- input: a set of node features, $\\mathbf{h} = \\{ \\bar{h}_1, \\bar{h}_2, \\dots, \\bar{h}_n \\}, \\quad \\bar{h}_i \\in \\mathbb{R}^{F}$\n",
    "- output: a set of node features, $\\mathbf{h'} = \\{ \\bar{h'}_1, \\bar{h'}_2, \\dots, \\bar{h'}_n \\}, \\quad \\bar{h'}_i \\in \\mathbb{R}^{F'}$\n",
    "- GAT\n",
    "    - by a parameterized linear transformation to every node\n",
    "        - $\\mathbf W\\cdot \\bar h_i, \\mathbf W\\in \\mathbf R^{F'\\times F}$\n",
    "    - self attention:\n",
    "        - $a: \\mathbf R^{F'}\\times \\mathbf R^{F'} \\rightarrow R$\n",
    "        - $e_{ij}=a(\\mathbf W\\cdot \\bar h_i,\\mathbf W\\cdot \\bar h_j)$：specify the Importance of node j's features  to node i\n",
    "    - normalization\n",
    "        - $\\alpha_{ij}=\\text{softmax}(e_{ij})=\\frac{\\exp(e_{ij})}{\\sum_{k\\in N(i)}\\exp(e_{ik})}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2eb409-d4b9-4deb-abb3-e2c8732dcecc",
   "metadata": {},
   "source": [
    "#### 认识 Cora 数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca1195ad-96a9-4ae2-95b2-cccffa050658",
   "metadata": {},
   "source": [
    "- https://github.com/kimiyoung/planetoid\n",
    "- Cora 是一个广泛使用的引用网络数据集，主要用于文献分类任务。它包含以下几个方面的信息：\n",
    "    - 节点（Nodes）：每个节点代表一篇学术论文。\n",
    "        - 特征（Features）：每个节点有一个高维的特征向量，通常是基于论文的词袋模型（Bag-of-Words）生成的。\n",
    "    - 边（Edges）：边表示论文之间的引用关系，即一篇论文引用了另一篇论文。\n",
    "    - 标签（Labels）：每篇论文被分配一个类别标签，表示其所属的研究领域，如机器学习、数据库、信息检索等。\n",
    "- 加载Cora数据集后，您可以进行以下操作：\n",
    "    - 节点分类：\n",
    "        - 使用图神经网络对论文进行分类，预测未标记论文的类别。\n",
    "    - 链接预测：预测论文之间是否存在引用关系。\n",
    "    - 图嵌入：学习节点的低维表示，以便在下游任务中使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ae8b29d-442b-41bf-94a2-b7676c676afe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:00.392255Z",
     "iopub.status.busy": "2024-10-27T14:42:00.391626Z",
     "iopub.status.idle": "2024-10-27T14:42:42.812745Z",
     "shell.execute_reply": "2024-10-27T14:42:42.811620Z",
     "shell.execute_reply.started": "2024-10-27T14:42:00.392195Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.x\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.tx\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.allx\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.y\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.ty\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.ally\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.graph\n",
      "Downloading https://github.com/kimiyoung/planetoid/raw/master/data/ind.cora.test.index\n",
      "Processing...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "from torch_geometric.data import Data\n",
    "from torch_geometric.nn import GATConv\n",
    "from torch_geometric.datasets import Planetoid\n",
    "import torch_geometric.transforms as T\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "name_data = 'Cora'\n",
    "dataset = Planetoid(root= './data/' + name_data, name = name_data)\n",
    "dataset.transform = T.NormalizeFeatures()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "04e31b70-fddb-40c9-9f50-908e4cfc8558",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:43:51.431899Z",
     "iopub.status.busy": "2024-10-27T14:43:51.430847Z",
     "iopub.status.idle": "2024-10-27T14:43:51.440465Z",
     "shell.execute_reply": "2024-10-27T14:43:51.439738Z",
     "shell.execute_reply.started": "2024-10-27T14:43:51.431851Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7, 1433, 0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.num_classes, dataset.num_node_features, dataset.num_edge_features, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8fdf6cd7-8d3d-433e-a79b-80545562fb29",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:48:48.475215Z",
     "iopub.status.busy": "2024-10-27T14:48:48.473791Z",
     "iopub.status.idle": "2024-10-27T14:48:48.487216Z",
     "shell.execute_reply": "2024-10-27T14:48:48.485740Z",
     "shell.execute_reply.started": "2024-10-27T14:48:48.475138Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(x=[2708, 1433], edge_index=[2, 10556], y=[2708], train_mask=[2708], val_mask=[2708], test_mask=[2708])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 边的属性如果有的话，叫 edge_attr??\n",
    "# x, y, edge_index, edge_attr\n",
    "dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d631f377-8a0e-4579-8f69-56d864c9f550",
   "metadata": {},
   "source": [
    "### coding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b96e7d8-df92-4fc8-9070-d1b226ed4453",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:42.813998Z",
     "iopub.status.busy": "2024-10-27T14:42:42.813667Z",
     "iopub.status.idle": "2024-10-27T14:42:42.818488Z",
     "shell.execute_reply": "2024-10-27T14:42:42.817484Z",
     "shell.execute_reply.started": "2024-10-27T14:42:42.813980Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5ca7971f-3e92-4b62-9974-313ab55d1df4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:42.819987Z",
     "iopub.status.busy": "2024-10-27T14:42:42.819670Z",
     "iopub.status.idle": "2024-10-27T14:42:42.900681Z",
     "shell.execute_reply": "2024-10-27T14:42:42.899732Z",
     "shell.execute_reply.started": "2024-10-27T14:42:42.819960Z"
    }
   },
   "outputs": [],
   "source": [
    "class GATLayer(nn.Module):\n",
    "    \"\"\"\n",
    "    Simple PyTorch Implementation of the Graph Attention layer.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        super(GATLayer, self).__init__()\n",
    "      \n",
    "    def forward(self, input, adj):\n",
    "        print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9188e735-e35b-4d6c-a648-3e0b5affa4e9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:42.902630Z",
     "iopub.status.busy": "2024-10-27T14:42:42.902212Z",
     "iopub.status.idle": "2024-10-27T14:42:42.999808Z",
     "shell.execute_reply": "2024-10-27T14:42:42.997934Z",
     "shell.execute_reply.started": "2024-10-27T14:42:42.902597Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([3, 5]), torch.Size([5, 2]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "in_features = 5\n",
    "out_features = 2\n",
    "nb_nodes = 3\n",
    "\n",
    "X = torch.rand(nb_nodes, in_features) \n",
    "W = nn.Parameter(torch.zeros(size=(in_features, out_features))) #xavier paramiter inizializator\n",
    "nn.init.xavier_uniform_(W.data, gain=1.414)\n",
    "\n",
    "X.shape, W.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8eb7e037-ca1c-481f-a827-caaf431015f0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.008117Z",
     "iopub.status.busy": "2024-10-27T14:42:43.007236Z",
     "iopub.status.idle": "2024-10-27T14:42:43.093182Z",
     "shell.execute_reply": "2024-10-27T14:42:43.091262Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.008053Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = torch.mm(X, W)\n",
    "N = h.shape[0]\n",
    "h.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3fad9092-2a5d-4929-88b5-9b445643eb38",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.096152Z",
     "iopub.status.busy": "2024-10-27T14:42:43.095369Z",
     "iopub.status.idle": "2024-10-27T14:42:43.201124Z",
     "shell.execute_reply": "2024-10-27T14:42:43.199480Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.096087Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1179, -0.3074],\n",
       "        [-0.1179, -0.3074],\n",
       "        [-0.1179, -0.3074],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.2360,  0.6803],\n",
       "        [ 1.2360,  0.6803],\n",
       "        [ 1.2360,  0.6803]], grad_fn=<ViewBackward0>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h.repeat(1, N).view(N*N, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "75597ccf-3bbe-4d4f-bc21-474ae6882449",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.204610Z",
     "iopub.status.busy": "2024-10-27T14:42:43.203372Z",
     "iopub.status.idle": "2024-10-27T14:42:43.292610Z",
     "shell.execute_reply": "2024-10-27T14:42:43.291323Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.204543Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1179, -0.3074],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.2360,  0.6803],\n",
       "        [-0.1179, -0.3074],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.2360,  0.6803],\n",
       "        [-0.1179, -0.3074],\n",
       "        [ 1.5600,  0.5294],\n",
       "        [ 1.2360,  0.6803]], grad_fn=<RepeatBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h.repeat(N, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7b6a9bf6-5b64-4fb5-a9a9-a2ed37980c16",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.294820Z",
     "iopub.status.busy": "2024-10-27T14:42:43.294332Z",
     "iopub.status.idle": "2024-10-27T14:42:43.380845Z",
     "shell.execute_reply": "2024-10-27T14:42:43.379187Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.294779Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1179, -0.3074, -0.1179, -0.3074],\n",
       "        [-0.1179, -0.3074,  1.5600,  0.5294],\n",
       "        [-0.1179, -0.3074,  1.2360,  0.6803],\n",
       "        [ 1.5600,  0.5294, -0.1179, -0.3074],\n",
       "        [ 1.5600,  0.5294,  1.5600,  0.5294],\n",
       "        [ 1.5600,  0.5294,  1.2360,  0.6803],\n",
       "        [ 1.2360,  0.6803, -0.1179, -0.3074],\n",
       "        [ 1.2360,  0.6803,  1.5600,  0.5294],\n",
       "        [ 1.2360,  0.6803,  1.2360,  0.6803]], grad_fn=<CatBackward0>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c8b812d0-d089-42bf-888e-b5ef6fce1f7f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.383149Z",
     "iopub.status.busy": "2024-10-27T14:42:43.382616Z",
     "iopub.status.idle": "2024-10-27T14:42:43.467308Z",
     "shell.execute_reply": "2024-10-27T14:42:43.466036Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.383106Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.1179, -0.3074, -0.1179, -0.3074],\n",
       "         [-0.1179, -0.3074,  1.5600,  0.5294],\n",
       "         [-0.1179, -0.3074,  1.2360,  0.6803]],\n",
       "\n",
       "        [[ 1.5600,  0.5294, -0.1179, -0.3074],\n",
       "         [ 1.5600,  0.5294,  1.5600,  0.5294],\n",
       "         [ 1.5600,  0.5294,  1.2360,  0.6803]],\n",
       "\n",
       "        [[ 1.2360,  0.6803, -0.1179, -0.3074],\n",
       "         [ 1.2360,  0.6803,  1.5600,  0.5294],\n",
       "         [ 1.2360,  0.6803,  1.2360,  0.6803]]], grad_fn=<ViewBackward0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * out_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5522b15b-0faf-42e1-a716-c8ad88a97d1c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.470258Z",
     "iopub.status.busy": "2024-10-27T14:42:43.469057Z",
     "iopub.status.idle": "2024-10-27T14:42:43.548645Z",
     "shell.execute_reply": "2024-10-27T14:42:43.547016Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.470216Z"
    }
   },
   "outputs": [],
   "source": [
    "a_input = torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * out_features)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "069552f8-35e8-4e11-b2fa-287d50caf3ee",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.550896Z",
     "iopub.status.busy": "2024-10-27T14:42:43.550396Z",
     "iopub.status.idle": "2024-10-27T14:42:43.635266Z",
     "shell.execute_reply": "2024-10-27T14:42:43.633884Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.550855Z"
    }
   },
   "outputs": [],
   "source": [
    "leakyrelu = nn.LeakyReLU(0.2)  # LeakyReLU\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "23c1081a-8a01-4c3f-9966-bffa0640d56f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.638036Z",
     "iopub.status.busy": "2024-10-27T14:42:43.637043Z",
     "iopub.status.idle": "2024-10-27T14:42:43.762953Z",
     "shell.execute_reply": "2024-10-27T14:42:43.761225Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.637994Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 1])\n"
     ]
    }
   ],
   "source": [
    "a = nn.Parameter(torch.zeros(size=(2*out_features, 1))) #xavier paramiter inizializator\n",
    "nn.init.xavier_uniform_(a.data, gain=1.414)\n",
    "print(a.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4cec1e07-94f6-46ef-8df7-21e408ae6c89",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-27T14:42:43.765329Z",
     "iopub.status.busy": "2024-10-27T14:42:43.764812Z",
     "iopub.status.idle": "2024-10-27T14:42:43.851404Z",
     "shell.execute_reply": "2024-10-27T14:42:43.850008Z",
     "shell.execute_reply.started": "2024-10-27T14:42:43.765289Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([3, 3, 4]) torch.Size([4, 1])\n",
      "\n",
      "torch.Size([3, 3, 1])\n",
      "\n",
      "torch.Size([3, 3])\n"
     ]
    }
   ],
   "source": [
    "print(a_input.shape,a.shape)\n",
    "print(\"\")\n",
    "print(torch.matmul(a_input,a).shape)\n",
    "print(\"\")\n",
    "print(torch.matmul(a_input,a).squeeze(2).shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b87fc360-8cae-4715-b24d-53381b91fa5e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
